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•  Outline 
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•  WG  3  Objectives 

•  Objective  1 :  Understand  the  impact  of  the  application  of 
traditional  operational  research  techniques  to  networked  C2 
systems. 

•  Objective  2:  Develop  inputs  to  the  C2  Metrics  Framework  for 
networked  C2  systems  and  “systems  of  systems”  to  measure 
and  assess  network  behaviors. 

•  Objective  3:  Identify  and  categorize  families  of  C2  measures  of 
effectiveness  useful  for  networked  C2  systems. 
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•  Bottom  Line  Up  Front 

•  The  SoS  C2  Network  perspective  is  different  from  the  System 
perspective  because  of  how  the  SoS  is  planned,  developed, 
integrated  &  tested,  and  operated. 

•  The  SoS  C2  Network  conveys  C2  Information  across  a  variety  of 
media  and  spectrum  from  highly  technical  machine-oriented 
methods  and  tools  to  very  social  human  interaction. 

•  Analysis  of  a  SoS  C2  Network  is  different  from  the  analysis  of  a 
system  in  how  you  have  to  plan  and  scope  the  analysis  and  not 
in  the  methods  and  tools  used 
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•  Definitions 

•  System  -  A  functionally,  physically,  and/or  behaviorally  related 
group  of  regularly  interacting  or  interdependent  elements;  that 
group  of  elements  forming  a  unified  whole  [JP  1-02  &  JP  3-0, 
DOD  SE  Guide  to  SoS]. 

•  System  of  Systems  -  An  SoS  is  defined  as  a  set  or  arrangement 
of  systems  that  results  when  independent  and  useful  systems 
are  integrated  into  a  larger  system  that  delivers  unique 
capabilities  [DoD,  2004(1)].  Both  individual  systems  and  SoS 
conform  to  the  accepted  definition  of  a  system  in  that  each 
consists  of  parts,  relationships,  and  a  whole  that  is  greater  than 
the  sum  of  the  parts;  however,  although  an  SoS  is  a  system,  not 
all  systems  are  SoS. 
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•  In  DoD  and  elsewhere,  SoS  can  take  different  forms.  Based  on  a 
recognized  taxonomy  of  SoS,  there  are  four  types  of  SoS  which  are 
found  in  the  DoD  today  [Maier,1998;  Dahmann,  2008]. 

•  Virtual.  Virtual  SoS  lack  a  central  management  authority  and  a  centrally  agreed  upon  purpose  for 
the  system -of-systems.  Large-scale  behavior  emerges — and  may  be  desirable — but  this  type  of  SoS 
must  rely  upon  relatively  invisible  mechanisms  to  maintain  it. 

•  Collaborative.  In  collaborative  SoS  the  component  systems  interact  more  or  less  voluntarily  to 
fulfill  agreed  upon  central  purposes.  The  Internet  is  a  collaborative  system.  The  Internet  Engineering 
Task  Force  works  out  standards  but  has  no  power  to  enforce  them.  The  central  players  collectively 
decide  how  to  provide  or  deny  service,  thereby  providing  some  means  of  enforcing  and  maintaining 
standards. 

*  Acknowledged.  Acknowledged  SoS  have  recognized  objectives,  a  designated  manager,  and 
resources  for  the  SoS;  however,  the  constituent  systems  retain  their  independent  ownership, 
objectives,  funding,  and  development  and  sustainment  approaches.  Changes  in  the  systems  are 
based  on  collaboration  between  the  SoS  and  the  system. 

*  Directed.  Directed  SoS  are  those  in  which  the  integrated  system -of-systems  is  built  and  managed 
to  fulfill  specific  purposes.  It  is  centrally  managed  during  long-term  operation  to  continue  to  fulfill 
those  purposes  as  well  as  any  new  ones  the  system  owners  might  wish  to  address.  The  component 
systems  maintain  an  ability  to  operate  independently,  but  their  normal  operational  mode  is 
subordinated  to  the  central  managed  purpose. 

Maier,  M.  (1998);  "Architecting  Principles  for  Systems-of-Systems";  Systems  Engineering,  Vol.  1,  No.  4  (pp  267-284). 

Dahmann,  Judith  and  Kristen  Baldwin,  (2008),  “Understanding  the  Current  State  of  US  Defense  Systems  of  Systems  and  the  Implications  for  Systems  Engineering”, 
Montreal,  Canada:  IEEE  Systems  Conference,  7-10  April. 
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•  Why  is  analysis  of  an  SoS  different  than  a  System? 

•  Complex  Stakeholder  Base 

•  SoS  Stakeholder  is  not  necessarily  the  same  as  the  System 
Stakeholders 

•  Independent  Goals  of  Systems  (e.g.,  missions  other  than  the 
SoS  Mission) 

•  Independent  Operation  and  Governance  of  Systems 

•  SoS  Analysis  includes: 

•  Aggregation  of  System  Analyses 

•  Additional  Analysis  at  the  SoS  level 

•  A  complex  Analysis  problem 
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•  Challenges  (may  affect  measurement,  methods  or  tool) 

(Understanding  the  differences  between  analyzing  a  System  or  SoS) 

•  Measuring  C2  Network  Effectiveness 

•  Must  be  done  in  context  of  Problem(s)  or  Mission(s) 

•  Systems  may  have  non-SoS  Missions 

•  Socio-Technical  Issues 

•  Acceptance  of  results  by  multiple  stakeholders 

•  System  data  releasibility  and  availability  to  the  SoS  level 

•  Measurement  (where  in  the  SoS) 

•  At  the  interfaces  between  systems 

•  Within  the  constituent  systems 

•  Human  actions  vs.  System/SoS  automated/machine  actions 

•  Frame  of  reference  for  human  performance  in  SoS  context  vs. 
the  system  context 
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•  Challenges 

•  Analyzing  the  contribution  of  SoS  C2  Network  on  Mission 
Outcome 

•  Human  decision-makers  act  inside  and/or  outside  the  system 
and  SoS  C2  Network  and  Processes 

•  More  Open  SoS  -  Creative/Manual  Processes 

•  More  Closed  SoS  -  Rule-based/Automated  Processes 
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•  OR  Techniques  for  Networked  C2  Systems  (1  of  2) 

•  OR  Techniques  should  be  applied  by: 

•  C2  Framework  Layer  (Social,  Cognitive,  Information,  Physical) 

•  Applicability  to  Mission,  Task 

•  Method/Tool 

•  Difficulty  in  Achieving  useful  Result 

•  SoS  Life-Cycle  Environment  (orientation  to  JCIDS) 

•  Development,  Integration, Test, Operations 

•  Traditional  Process  (based  on  WG1  &  past  literature) 

1 .  Characterize  System 

2.  Define  Objectives  and  Goals  of  System 

3.  Define  the  Required  System  Analysis 

4.  Identify  the  Measures  of  Merit  (MoMs) 

5.  Apply  OR  Tools 
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•  OR  Techniques  for  Networked  C2  Systems  (2  of  2) 

•  SoS  Process 

1 .  Characterize  SoS 

•  Interdependencies  -  interoperability,  connectivity,  resiliency, 
redundancy,  security,  etc. 

•  System  Types  -  level  of  autonomy  (deterministic,  stochastic, 
chaotic,  etc.) 

•  Authority/Ownership  -  control,  physical,  operational, 
technical,  financial,  governance 

2.  Define  the  System  and  SoS-unique  Analysis 

3.  Define  Objectives  and  Goals  of  Systems  and  SoS 

4.  Identify  the  Measures  of  Merit  (MoMs) 

5.  Apply  OR  Tools 

•  Necessary  Elements  for  Analysis: 

•  Metrics,  Mission,  Architecture,  Scenarios 
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•  Example  of  SoS  Metrics 


OR  Technique 

C2  Network  Layers 

Metric 

Framework 

Characteristic 

SOS  Metric 

NATO 

category 

Sub  Metric 

Decision  Theory 

Game  Theory 

Queuing  Theory 

Modeling  (static) 

Statistical  Analysis 

Optimization 

techniques 

M&S/  Dynamic 

Simulation 

Network  Theory 

Reliability  Theory 

Physical 

Information 

Cognitive 

Social 

C2  SOS 

Interoperability 

Connectivity 

MOCE 

X 

X 

X 

X 

Clarity 

X 

X 

X 

X 

Completeness 

X 

X 

X 

X 

X 

Data  Assurance, 
Integrity 

X 

X 

X 

Service  Demand 

X 

X 

X 

X 

X 

Interdependency 

Timeliness 
(staleness,  delay, 
latency) 

X 

X 

Responsiveness  (to 
command,  info 
need) 

X 

X 

X 

X 

X 

Connectivity 

X 

X  . 

k 

u 

no  id: 

5SI1 

ieu 

PO  Command 
Control 

•  ■'’.■"C1  O’  001 

MORS  Workshop:  A  Joint  Framework  for  Measuring 

Command  and  Control  Effectiveness 

u  QOi 

Unclassified 


•  Summary 

•  Traditional  OR  techniques  may  not  change  but  how  they  are 
applied  can  change  based  on  the  scope  of  the  C2  Network. 

•  Non-traditional  analysis  techniques  may  be  required  to  account 
for  the  effect  of  the  decision-makers  interacting  with  C2  Network 
Systems  and  SoS. 

•  In  general,  System-type  metrics  apply  at  the  SoS-level,  but  there 
are  additional  levels  of  complexity  in  applying  them  to  the 
problem  and  they  reveal  different  characteristics  about  the  SoS 

•  Identification  of  SoS  Mission/Goals,  Architectures,  Scenarios  are 
necessary  to  determine/derive  appropriate  metrics 

•  Measures  of  Effectiveness  do  not  always  relate  directly  to 
Measures  of  Performance 

•  Multiple  Stakeholders  more  likely  in  SoS,  and  may  affect  the 
acceptance  of  SoS  results 
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•  Recommendations 

•  Frame  the  discussion  before  breaking  into  the  Working  Groups 

•  Integration  of  C2  Metrics  Framework  across  the  WG 
Persceptives 
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•  Back-up  Material 
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Aspect  of 
Environment 

Acknowledged  SoS* 

Virtual  SoS 

Collaborative  SoS 

Directed  SoS 

Management  &  Oversight 

Stakeholder 

Involvement 

Stakeholders  at  both  system 
level  and  SoS  levels  (including 
the  system  owners),  with 
competing  interests  and 
priorities;  in  some  cases,  the 
system  stakeholder  has  no 
vested  interest  in  the  SoS;  all 
stakeholders  may  not  be 
recognized. 

A  virtual  SoS  has  no  centrally 
established  purposes  but  rather  the 
puipose  expresses  itself  as  the 
collective  actions  of  the  individual 
systems. 

Stakeholders  negotiate  among 
themselves  to  establish  a  common 
puipose.  The  SoS  is  built  to  this 
purpose  and  the  individual  systems 
negotiate  among  themselves  to 
determine  which  part  of  this 
responsibility  each  fulfills.  Central 
players  often  establish  the  ground  rules 
by  which  other  players  participate. 

A  central  SoS  authority  usually 
establishes  the  purpose  to  be 
achieved  by  the  SoS.  The  SoS 
is  built  to  this  purpose  and  the 
individual  systems  are 
generally  directed  by  the 
central  authority. 

Governance 

Added  levels  of  complexity 
due  to  management  and 
funding  for  both  the  SoS  and 
individual  systems;  SoS  does 
not  have  authority  over  all  the 
systems. 

No  central  body  controls  the  purpose 
or  management  of  the  SoS  or 
individual  systems.  Governance  may 
emerge  from  politics  or  policies 
agreed  to  by  stakeholders  but  none  is 
compelled  to  comply. 

In  collaborative  SoS  there  is  no  central 
authority  with  the  power  to  enforce  a 
particular  SoS  purpose.  A  central 
authority  may  establish  purposes, 
standards,  etc.,  which  are  usually 
complied  with,  but  does  not  have 
authority  to  enforce  them. 

Individual  systems  are 
governed  by  membership  to  a 
common  SoS  command 
structure  which  usually 
includes  a  central  governing 
authority. 

Operational  Environment 

Operational 

Focus 

Called  upon  to  meet  a  set  of 
operational  objectives  using 
systems  whose  objectives  may 
or  may  not  align  with  the  SoS 
objectives. 

Individual  systems  are  operated 
independently.  Operation  of  the  SoS 
is  complex  because  there  is  no 
centrally  directed/controlled  puipose. 
Participation  by  systems  is  voluntary 
and  they  often  have  conflicting 
purposes  which  they  will  try  to  attain 
simultaneously  with  other  systems. 

Collaborative  SoS  differs  from  directed 
SoS  in  that  a  central  authority  is  not 
able  to  enforce  particular  operation  of 
the  system.  Systems  collaborate  of  their 
own  will  to  achieve  a  central  purpose; 
however,  from  time  to  time  SoS 
operational  needs  are  subjugated  to  the 
needs  of  a  particular  system. 

The  systems  are  connected  by 
command  and  control 
structures.  The  SoS  directs  the 
operation  of  individual  systems 
to  achieve  the  SoS  purpose  (a 
centralized  control  authority). 
Systems  are  usually  allowed 
operational  independence  to 
deal  with  local  situations. 

Table  adapted  &  Acknowledged  SoS  definitions  from  DoD  SE  Guide  for  SoS;  others  defined  by  Clyde  Smithson. 
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Aspect  of 
Environment 

Acknowledged  SoS* 

Virtual  SoS 

Collaborative  SoS 

Directed  SoS 

Implementation 

Acquisition 

Added  complexity  due  to 
multiple  system  lifecycles 
across  acquisition  programs, 
involving  legacy  systems, 
systems  under  development, 
new  developments,  and 
technology  insertion;  Typically 
have  stated  capability 
objectives  upfront  which  may 
need  to  be  translated  into 
formal  requirements. 

Component  systems  are  acquired 
independently  without  regard  to  other 
systems,  except  in  the  context  that 
another  system  may  perform  a 
beneficial  function  for  that  system 
(usually  at  little  or  no  cost)  and 
dependably. 

Systems  negotiate  among  themselves  to 
determine  how  SoS  objectives  are  to  be 
met  and  which  system  is  to  provide 
which  SoS  capability.  Agreements  are 
made  between  central  players  to  form  a 
common  acquisition  strategy.  This  can 
be  seen  as  negotiated  “political” 
objective  as  opposed  to  direction  by 
central  authority. 

Individual  systems  are 
acquired  through  different 
program  offices  and  operated 
separately;  however,  there  is  a 
central  authority  directing, 
coordinating,  and  balancing  the 
various  program  offices. 
Systems  may  be  custom  built 
to  meet  the  needs  of  the  SoS. 

Test& 

Evaluation 

Testing  is  more  challenging 
due  to  the  difficulty  of 
synchronizing  across  multiple 
systems’  life  cycles;  given  the 
complexity  of  all  the  moving 
parts  and  potential  for 
unintended  consequences. 

SoS  testing  generally  occurs  on  an  ad 
hoc  basis.  Individual  systems  test 
themselves.  Testing  at  the  SoS  level 
is  confined  to  aspects  of  the  SoS  at 
that  level  that  affect  the  function  and 
purpose  of  individual  systems.  In 
other  words  a  system  only  tests  what 
is  important  to  itself  at  the  SoS  level, 
if  any  SoS  testing  is  conducted  at  all. 

SoS  testing  is  established  by 
coordination  and  negotiation  between 
the  central  SoS  players.  Testing  tends 
to  change  over  time  as  the  SoS  purpose 
evolves.  For  a  directed  SoS  the  testing 
tends  to  be  directed  from  top  down 
whereas  for  virtual  SoS  it  springs  up 
organically.  T&E  for  a  collaborative 
system  comes  from  a  middle  ground  in 
which  the  central  players  establish  goals 
that  are  tested  by  the  entire  SoS. 

Testing  occurs  at  multiple 
levels  but  is  directed  from  the 
SoS  level  At  the  SoS  level 
testing  is  directed  to  evaluate 
the  central  purpose  of  the  SoS. 
Testing  may  occur  with  the 
entire  SoS  or  portions  of  it. 
Additionally,  testing  occurs  at 
the  system  level  to  establish 
that  the  system  meets  its 
individual  requirements, 
including  those  supporting  the 
system  purpose. 

Table  adapted  &  Acknowledged  SoS  definitions  from  DoD  SE  Guide  for  SoS;  others  defined  by  Clyde  Smithson. 
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Aspect  of 
Environment 

Acknowledged  SoS* 

Virtual  SoS 

Collaborative  SoS 

Directed  SoS 

Engineering  &  Design  Considerations 

Boundaries 

and 

Interfaces 

Focus  on  identifying  the 
systems  that  contribute  to  the 
SoS  objectives  and  enabling 
the  flow  of  data,  control  and 
functionality  across  the  SoS 
while  balancing  needs  of  the 
systems. 

Boundaries  and  interfaces  evolve 
through  adaptation  and  survival  - 
successful  standards  live  and  are 
extended  upon  while  others  die  out. 
Forces  other  than  the  technical  merits 
of  these  may  determine  survival  (e.g., 
VHS  vs.  Betamax).  Systems  choose 
to  use  or  not  use  these  at  their  own 
discretion.  A  standard  may  be  created 
by  an  individual  system,  and  then  be 
adopted  by  others. 

Certain  systems  rise  to  be  central 
players  at  the  SoS  level.  These  systems 
usually  reach  agreement  on  what  the 
interface  standards  are  and  what 
services  to  provide.  They  usually  create 
common  standards  for  use  by  the  entire 
SoS  but  do  not  enforce  them  (except  by 
operationally  excluding  other  systems 
that  do  not  conform). 

Interfaces  are  seen  as  a  key 
integrating  factor  for  the  SoS. 

A  central  authority  establishes 
the  interface  requirements, 
with  input  from  the  component 
systems.  Similarly,  the  central 
authority  establishes  the 
boundaries  between  systems. 

Performance 
&  Behavior 

Performance  across  the  SoS 
that  satisfies  SoS  user 
capability  needs  while 
balancing  needs  of  the 
systems. 

The  performance  of  the  SoS  is  not 
directed,  but  rather  is  an  emergent 
behavior.  There  are  no  established 

SoS  performance  requirements. 
Individual  systems  optimize  to 
perform  best  for  their  own  ends  (i.e., 
best  ROI  at  the  system  level)  and  SoS 
performance  derives  from  that. 

Like  the  virtual  SoS,  there  are  no 
minimum  SoS  performance 
requirements  enforced  by  a  central 
authority.  Rather,  the  constituent 
systems  agree  to  a  set  of  mutual 
performance  goals  and  behaviors  which 
evolve  over  time.  Individual  systems 
may  choose  to  sub  optimize  to  benefit 
the  SoS. 

All  constituent  systems  must 
met  minimum  performance 
requirements  to  satisfy  SoS 
capability  requirements. 
Individual  systems  may  be 
operated  sub  optimally  to  meet 
SoS  performance  requirement. 
Generally,  individual  system 
performance  is  secondary  to 

SoS  performance. 

Table  adapted  &  Acknowledged  SoS  definitions  from  DoD  SE  Guide  for  SoS;  others  defined  by  Clyde  Smithson. 
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•  SOS  Viewpoints 

•  SOS  -  similar  to  “systems” 

•  but  component  Systems  are  non-dedicated 

•  Decomposition  applies 

•  Interfaces,  information  flow  concepts  are  similar  to 
Systems  analysis 


o 

ooo 
0  0 


•  Analysis  of  SOS  Interdependencies,  control,  autonomy 


C2 


Command 
&  Control 
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•  WG  3  Findings 

•  Obj  1 

•  Obj  2 

•  Obj  3 

•  Noteworthy  discussion  points 

•  Areas  for  further  research 
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•  WG  Recommendations 

•  Should  be  actionable,  linked  to  findings 

•  Write  or  re-write  regulation  or  instruction 

•  Research  analytical  approaches 

•  Build  or  improve  a  model 

•  Sponsor  a  study 

•  DOTMLPF 

•  Identify  OPR 

•  MORS  can  be  OPR  if  recommendation  is  in  the  Society’s 
purview,  e.g.  form  a  MORS  COP,  hold  classified  workshop  or 
MORSS  Special  Session 

•  MORS  Sponsor 

•  TRADOC 

•  Other 
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•  C2  Network  Framework  Layers 

•  Physical 

•  Quality  of  Service 

•  Information 

•  Quality  of  Information 

•  Cognitive 

•  Quality  of  Decision 

•  Quality  of  Process 

•  Social 

•  Quality  of  Collaboration 
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•  Where  &  How  to  Apply  Metrics 

•  Heirarchy  of  metrics  in  the  context  of  the  SoS 

•  Policy  Questions 

•  Right  Force  Effectiveness  Metrics 

•  Right  Mission  Context 

•  Right  C2  Network  Metrics 
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•  Considerations  for  SoS  C2  Network  Analysis 

•  Metrics  used  to  characterize  the  C2  Network  should  be 
compared  to  mission  outcome/success  to  determine  their 
applicability  and  answer  such  questions: 

•  Did  more,  or  less,  Information  affect  the  outcome? 

•  What  was  the  Vital  Information? 

•  Did  the  Vital  Information  arrive  in  Time? 

•  Did  the  Vital  Information  reach  the  Decision  Maker  who 
needed  it? 
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•  Analytical  Methods  &  Tools 

•  Systems  Engineering  &  Architecture  tools 

•  Context  Diagrams,  Functional  Block  Diagrams, 
DODAF/MODAF,  business  process  models 

•  OR  Tools/Techniques 

•  Decision  Theory 

•  Game  Theory 

•  Queuing  Theory 

•  Modeling  (static) 

•  Statistical  Analysis 

•  Optimization  techniques 

•  M&S/  Dynamic  Simulation 

•  Network  Theory 

•  Reliability  Theory 
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•  SoS  C2  Network  Behavior 

•  Decision  Making 

•  The  degree  to  which  a  decision,  or  process,  is  centralized  to 
one  (or  a  subset)  of  systems  versus  distributed  across  the 
network. 

•  The  type  of  connectivity  used  ranging  from  machine  to  social 

•  Level  of  system  to  system  synchronization  required 
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•  SOS  Metrics  Table 
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•  Recommendations 

•  Stakeholder  buy-in  up-front  and  throughout  the  process  is  even 
more  critical  success  in  an  SoS  analysis 
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